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(57) Abstract 

A system for computer siqjported collaboration is provided wherein two aR>Ucations, having individual input devices, are coupled 
to a network which transmits aR)lication data between the two appUcatioos. Each application is capable of processing its own appUcaoon 
data and die application data of the application at the remote end of the nctwcnk snnultaneously. 
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System for Computer Supported Collaboration 

BacXaround of the Invention 

1> Field of the Invention 

This invention relates to the field of video processing 
and, in particular, to the processing of video audio and data 
signals which are transmitted between computers. 

2 > Baclcaround Art 

Video conferencing networks may be isochronous networks 
having a fixed bandwidth and a fixed delay. This permits trans- 
mitted video compressed at one end of a video conferencing 
network to arrive at the remote end of the network in a fixed 
fashion so that it can be smoothly decompressed and displayed. 
This also prevent a choppy reproduction. 
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The problems caused by choppy video reproduction may 
not be very noticeable. However, when audio reproduction is 
choppy the results are unsatisfactory- The only isochronous 
network available at this time for avoiding this problem is the 
telephone system. 

Summary of the Invention 

The computer supported collaboration system of the 
present invention includes both video and audio data communica- 
tions. The communication of video and audio data are simulta- 
neous with the sharing of application programs, and their appli- 
cation program data, by way of a connection between computers. 
The connection between computers may be implemented in a number 
of ways. For example, the public telephone network may provide a 
connection between computers which are located in different 
locations, in a single country, or in locations in different 
countries. However, the inventive system is not limited to 
connections by way of public telephone networks. For example the 
connection may be a dedicated data connection between the 
computers . 

A video conference application program allows an 
initiating user to select a remote computer using the name of the 
destination user or using a corresponding identifier such as the 
telephone number of the destination user. A video conference may 
then be established with the destination user. The application 
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program of the initiating user places a call to the selected 
remote computer. This may be performed using a CALL button that, 
when pressed, presents the initiating user with a listing of 
names and identifiers. If the remote computer is also running 
the video conference application program the initiating applica- 
tion program begins the video conference by transmitting com- 
pressed interleaved audio and video frames to the remote computer 
by way of the data connection. The remote computer also starts 
transmitting compressed interleaved audio and video frames. When 
each of the computers receives data the compressed data is decom- 
pressed and displayed or heard in the case of audio. 

Images of the users of the computer supported collabo- 
ration system may be displayed in a small window on their own 
displays in addition to being displayed at the other end of the 
system. This permits users to determine that the image received 
by the remote user is suitable for display. The remote user is 
preferably displayed in a larger window on each display in order 
to keep the focus of the users on the remote users. A HANGUP 
button may also be provided for use when a video conference is in 
progress. This permits termination of the video conference. 

The sharing of application programs between the remote 
terminals and the sharing of their data in- the computer supported 
collaboration system of the present invention may be performed 
using various data sharing embodiments. A very general data 
sharing embodiment is an embodiment wherein mouse drivers and 
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keyboard drivers are provided in order to: (1) send data from 
both the mouse and the keyboard to the remote computer operating 
system on the remote side of the network, as well as (2) send 
data to the operating system of the local computer. This embodi- 
ment permits any application program to be shared across the 
network even if it was not originally designed to be shared. 

An alternate data sharing embodiment is one wherein an 
application program is written in a manner which adapts it for 
data sharing during a video conference- In this data sharing 
embodiment each application program transmits its data to the 
remote application program in an appropriate manner. A further 
data sharing embodiment is one wherein the operating system of 
each of the computers allows application programs to cut and/ or 
copy data into a respective clipboard which is effectively shared 
by the two computers. The shared clipboard of each computer 
transmits this data to the shared clipboard of the remote 
computer. This permits any application which provides cut and/ or 
copy capability to be shared using the computer supported collab- 
oration system of the present invention. 

Brief Description of the Dravinas 

Fig. 1 shows a block diagram representation of the 
computer supported collaboration system of the present invention. 
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Fig. 2 shows a block diagram representation of a data 
sharing embodiment which may be implemented on the computer 
supported collaboration system of Fig 1. 

Fig. 3 shows a block diagram representation of an 
alternate data sharing embodiment which may be implemented on the 
computer supported collaboration system of Fig. 1. 

Fig. 4 shows a block diagram representation of a 
further alternate data sharing embodiment which may be imple- 
mented on the computer supported collaboration system of Fig. 1. 

Fig. 5 shows a block diagram of a video conference 
portion of the computer supported collaboration system of Fig. 1. 

Detailed Description of the Tnvention 

Referring now to Fig. 1, there is shown video con- 
ferencing system 100 wherein application programs 114, 120 run 
simultaneously on personal computers 102, 104 and communicate 
with each other. The communication between personal computers 
102, 104, executing application programs 114, 120, is by way of 
video conferencing network 124. In this embodiment of the 
present invention a video teleconference is supported between 
personal computers 102, 104 wherein the video images and the 
audio signals generated at the end of video conferencing system 
100 where personal computer 102 is disposed are captured in 
computer 102. The captured signals in personal computer 102 are 



-5- 



wo 94/24629 



PCT/US94/03960 



transmitted by way of video conferencing network 124 to personal 
computer 104. At personal computer 104 the video captured by 
personal computer 102 is displayed and the audio is played out. 

Depending on the type of data sharing implemented on 
video conferencing system 100 application programs 114, 120 may 
be designed for data sharing with a number of users or they may 
be completely independent of any previous data sharing 
capability. For example, applications programs 114, 120 may be 
Microsoft Excel. Using video conferencing system 100 application 
programs 114, 12 0 operating upon personal computers 102, 104 may 
share network resources. In conferencing system 100 the audio 
data streams as well as the video data streams of personal 
computers 102, 104 are multiplexed over bidirectional data 
transmission line 138. The operations and functions of applica- 
tion collaboration drivers 127, 129 are described herein below 
for a variety of date showing embodiments. 

Referring now to Fig. 2, there is shown a block diagram 
flow representation of data sharing embodiment 200. Data sharing 
embodiment 200 may be implemented on the video conferencing sys- 
tem of the present invention using video conferencing 
network 224. Application program 214 of data sharing embodiment 
200, may receive its input both from keyboard 216 and mouse device 
230. The inputs from keyboard 216 and mouse device 2 30 are 
received and transmitted to application program 214 by way of 
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keyboard driver 2 34 and mouse driver 231 respectively • 
Application program 220 may receive its input from both keyboard 
222 and mouse device 228. The inputs from keyboard 222 and mouse 
device 228 are received and transmitted to application program 
220 by way of keyboard driver 23 6 and mouse driver 232 respec- 
tively- Drivers 231, 234 apply input data directly to applica- 
tion program 214. Drivers 232, 236 apply input data directly to 
application program 220. 

In a typical configuration of data sharing embodiment 
220, keyboards 216, 222 generate interrupts in response to a user 
depressing keys. In the preferred embodiment keyboard drivers 
234, 236 handle these interrupts themselves in order to receive 
the information from keyboards 216, 222 and apply it to applica- 
tion programs 214, 220. Drivers 231, 234 may be software rou- 
tines adapted to apply data to application program 214 by way of 
bidirectional data transmission lines 240, 242. In data confere- 
ncing system 200 drivers 231, 234 also apply keyboard data and 
mouse device data to data sharing driver 250. When data sharing 
driver 250 receives this data from bidirectional data transmis- 
sion lines 240, 242, it applies the received data to a remote 
computer by way of video conferencing network 224. 

Similar to the manner described with respect to drivers 
231, 234, drivers 232, 236 may be software routines adapted to 
apply data to an application program such as application program 
220. Drivers 232, 236 apply this data to application program 220 
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by way of bidirectional data transmission lines 244, 246. The 
data applied to application program 220 is simultaneously applied 
to data sharing driver 260 by way of data transmission lines 244, 
246, Data sharing driver 260 applies the data received by way of 
transmission lines 244, 246 to video conferencing network 224 for 
transmission to a remote computer. For example, the data applied 
to network 224 by drivers 232, 236 may be received by data 
sharing driven 2 50 for use and display by application program 214 
under the control of data sharing application 252. 

Thus, input into either application program 214, 220 is 
shared with the other application program 214, 220 within data 
sharing embodiment 2 00. In order to share data in this manner 
application program 214 and application program 220 should be the 
same revision of the same application. Additionally, the. screen 
arrangement for the display of application programs 214, 220 
should be exactly the same. For example, when working on a 
certain cell of a spreadsheet in one application program 214, 
220 the screen should be set up the same way in the other appli- 
cation program 214, 220 so that the same cell is modified by 
movements of mouse devices 228, 230. Even though application 
programs 214, 220 are operating exactly in parallel in this 
manner, the fact that they are being shared is completely trans- 
parent to their operation and each operates as if the other is 
not operating. 
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It is the operation of data sharing applications 252, 
262 which permits the data sharing of data sharing embodiment 
200. Data sharing applications 252, 262 control data sharing 
drivers 250, 260, respectively, by way of bidirectional data 
sharing control lines 251, 261. Under the control of data 
sharing drivers 250, 260 one application program 214, 220 at a 
time is the master of embodiment 200. The other application 
program 214, 220 is thus the slave during that time. During that 
time the display of the slave is driven by the input of the 
master. It will be understood that data flows only in one 
direction at a time through data transmission lines 240, 242, 
223, 225, 244, 246 even though they are bidirectional. 

In some operations it may not be convenient that 
application programs 214, 220 of embodiment 200 should be the 
same revision and that the screens should be set up the same way. 
However it is these restrictions which make it possible to 
synchronize application program 214 and application program 220 
and to perform the data sharing. When these restrictions can be 
accommodated an advantageous feature of data sharing embodiment 
200 is that application programs 214, 220 may be any existing 
application programs. Under these circumstances application 
programs 214, 220 do not have to be rewritten in order to permit 
data sharing between them within, data sharing citOaodiment 200. 

Furthermore, under these circumstances applications 
214, 220 may be almost any existing application. In order to 
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achieve this result data sharing applications 252, 260 are 
adapted to control and synchronize individual application 
programs 214, 220. This is accomplished by defining the inter- 
face between applications programs 214,220 and data sharing 
drivers 250,260. When application programs 214, 220 transmit 
data through data sharing applications 252, 262 data sharing 
drivers 250, 260 are able to direct the keyboard and mouse inputs 
to the proper locations for the remote application program 214, 
220. Data sharing applications are also able to issue commands 
to the remote application program 214, 220 to force it to conform 
to the restriction that screen set up and application set up are 
the same. In this capacity data sharing applications 252, 262 
can make sure that both application programs 214, 220 are editing 
or modifying the same file and working on the same data. 

Thus data sharing applications 252, 262 may be tailored 
for an operating system rather than for each set of application 
programs 214, 220. In a windowing type of operating system data 
sharing applications 252, 262 may make system calls to the 
operating system and the operating system responds in a well 
defined way. In non-windowed operating systems, it is somewhat 
more difficult to tailor data sharing applications 252, 262 to 
the operating system and they would have to be more tuned to the 
individual application programs 214, 220. 

Referring now to Fig. 3, there is shown a block diagram 
representation of data sharing embodiment 300. Data sharing 
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embodiment 3 00 may be implemented in the video conferencing sys- 
tem of the present invention using video conferencing network 
324, Video conferencing network 324 couples application programs 
314, 320 by way of data sharing drivers 350, 360 and bidirec- 
tional data transmission lines 323, 325. 

In data sharing embodiment 3 00 application programs 
314, 3 20 are written such that they are adapted to share data 
with each other. Thus when application program 314 receives 
input from mouse device 328 or keyboard 322 of remote application 
program 320 it is capable of processing it- Thus, in data 
sharing embodiment 300 both application programs 214, 320 are 
able to receive and interpret the inputs from their own keyboards 
316, 322 and mouse devices 323 , 330 as well as the inputs of the 
remote input devices 316, 322, 328, 330. For example, both 
application programs 314, 320 of data sharing embodiment 300 are 
able to determine whether to move an icon to a new location on 
their own screens and whether to execute a command in response to 
input from either keyboard 316, 322. 

Rather than transmitting raw inputs from mouse devices 
328, 330 or keyboards 316, 336 application programs 314, 330 send 
a set of commands through network 334 to remote application 
program 314, 330. These are application specific commands. 

For example, of application programs 314, 3 20 are 
spreadsheet programs and the user of application program 314 
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highlights a specific cell in the spreadsheet display. Applica- 
tion program 314 transmits a command indicating the cell high- 
lighting through video conferencing network 324 to remote 
application program 320. This command instructs application pro- 
gram 320 to highlight the corresponding cell on its own display. 
Application program 314, 320 of data sharing embodiment 300 are 
not required to be exact by the same revision as previously set 
froth with respect to data sharing embodiment 200. Furthermore, 
application program 314, 320 are not required to provide exact by 
the same screen display. Application programs 314, 320 are 
required, however, to be the same application and the data within 
application programs 314, 320 must be the same. 

Referring now to Fig. 4, there is shown a block diagram 
representation of data sharing embodiment 4 00. Data sharing 
embodiment 4 00 may be implemented on the video conferencing sys- 
tem of the present invention. within data sharing embodiment 400 
application programs 414, 420 interface with video conferencing 
network 424 by way of drivers 450, 460. Application programs 
414, 420 are designed to adhere to a set of application specific 
commands which are transmitted by way of video conferencing 
network 424 within data sharing embodiment 400. Application 
program 414 may be a spread sheet and application program 4 20 may 
could be a document. Alternately application program 414, 420 
may be the same application program. For example application 
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prograxa 414, 420 nvay both be spread sheets or they may both be 

documen'ts . 

Each application program 414, 420 of data sharing 
embodiment 400 runs its own respective shared clipboard 
application 470, 480. In this configuration application program 
414 may apply data to shared clipboard application 470 by way of 
line 468 using a cut or copy command. This data is simul- 
taneously transmitted through data sharing driver 450 and video 
conferencing network 424 to data sharing driver 460, shared 
Clipboard 480 and application program 420. When the transmitted 
data is received by application program 420 it may appear at 
Shared clipboard 480. When the data appears at shared clip- 
board 480 the user of application program 420 may issue a paste 
coxtonand. This permits application programs 414, 420 to share 
information using a cut and a copy as well as a paste on the 
remote side. 

Thus When one application program 414, 420 transmits 
data by way of video conferencing networK 424. The transmitting 
application 414. 420 performs a cut or copy and the other appli- 
cation program 414, 420 performs a paste operation. When the 
transmission arrives at the other remote application program 414, 

r-eeeivinci application 414, 420 
420, it just is dropped unless the recexvxng 

performs the paste operation. 
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Data sharing einbodiinent 400 avoids the problems 
associated with differing revisions of application programs as 
well as other possible incompatibilities between application 
programs which may arise within video conferencing system 100. 
Examples of the possible incompatibilities which may cause 
problems in other data sharing arrangements may be the way that 
the display screen is set up, differing drivers, and differing 
versions of the operating system. Data sharing embodiment 400 
thus allows most application programs having cut, copy and paste 
capabilities to perform the computer supported collaboration of 
the present invention. 

Referring now to Fig. 5, there is shown video con- 
ferencing embodiment 500. In video conferencing embodiment 500 
video data is received by way of video input 504 and audio data 
is received by way of audio input 508. Data received by inputs 
504, 508 is digitized by analog-to-digital converters 512. 516, 
respectively, to provide digitized video and analog input data 
streams. The digitized input data streams are applied to a 
compression engines 520, 524 within video conferencing embodiment 
500. 

in this manner a video input signal is received at 
video input 504, converted to digital by analog-to-digital 
converter 512, and applied to video compression engine 520 and an 
audio input signal is received at audio input 508, convert:ed to 
digital by analog-to-digital converter 516, and applied to audio 
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compression engine 524. In order to permit embodiment 500 to 
provide synchronization of the compressed outputs of compression 
engines 520, 524, engines 520, 524 generate frames of compressed 
video and audio, respectively. 

The frames of video produced by video compression 
engine 520 may be thirty frames per second or fewer. The video 
signal which is thus divided into separate compressed frames by 
compression engine 520 is applied to video frame buffer 528 one 
frame at a time. The compressed audio signal is sampled and, 
according to both the sample frequency of the audio and the 
number of video frames per second, the compressed audio data is 
broken up into frames of audio by audio compression engine 524 
and applied to audio frame buffer 532. 

The video frames of video frame buffer 528 and the 
audio frames of audio frame buffer 532 are interleaved by 
video/audio frame interleaving device 544. This process provides 
synchronization between the compressed video signals and the 
compressed audio signals. The frame interleaving performed by 

frame interleaving device 544 is a one-to-one interleaving of a 

frame of video data with a frame of audio data. 

The interleaved frames of data are applied to video 
conferencing network 552 by way of bidirectional data transmis- 
sion line 548 and transmitted by video conferencing network 552 
to the remote side of the system. It will be understood by those 
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skilled in the art that data transmission within video 
conferencing embodiment 500 through video conferencing network 
524 is bidirectional and that transmission in only one direction 
is shown in video conferencing embodiment 500 in order to 
simplify the drawing. 

On the receiving side of data sharing embodiment 500 
the interleaved audio and video frames from audio/video frame 
interleaving device 544 is stored buffer 556 wherein the inter- 
leaving audio and video frames are separated from each other. 
The separated data streams from buffer 556 are separately applied 
to video decompression engine 560 by way of line 559 and audio 
decompression engine 564 by way of line 563. Decompression 
engines 560, 564 are effective to undo the compression of com- 
pression engines 520, 524 on the transmitting side of video 
conferencing embodiment 500. 

The decompressed video data from video decompression 
engine 560 is applied to display device 568 for visual display 
within the receiving end of video conferencing embodiment 500. 
The decompressed audio data from audio decompression engine 564 
is applied to transducer devices such as speakers 572 to provide 
audio reproduction. These receiving and separating operations 
performed by buffer 566 and the receiving and decompressing 
operations of decompressing engines 560, 564, acting co- 
operatively with each other and with the compressing and inter- 
leaving operations of compression engines 520, 524, buffers 528, 
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532 and frame interleaving device 544, are effective to provide 
synchronization between the signals reproduced on display device 
568 and speakers 572. 

The computer supported collaboration system of the 
present invention was implemented using known lab/beta software 
packages such as those provided by Smart Technologies to incorpo- 
rate the sharing of application programs and their data in video 
conferencing using computer supported collaboration. The sharing 
of application programs in this implementation, as well as the 
sharing of data in computer supported collaboration in general, 
may be performed by writing mouse and keyboard drivers for 
Microsoft Windows 3.1 to perfonh the operations of drivers 231, 
232, 234, 236 in a manner understood by those skilled in the art. 
The characteristics of this computer supported collaboration 
implementation are set forth in Table I. 

Transfer Rate 256k bits/sec (full 

duplex) 

Video Algorithm RTV 2.0 

p™e Rate "P 12fps (b/c of 

Frame Rate inter-frame encoding) 

Video Data Rate Variable 

Delay (one way) tiaTJSo 

Spatial 128 x 120 

Resolution 

Audio Algorithm ADPCM4E 

sample Rate 8K samples/sec 

Audio Data Rate 32K bits/sec 

TABLE I 
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A suitable operating system for providing video confer- 
encing system 100 in this manner may be DOS 5.0, Programs which 
provide suitable windowing capability are well known in the art. 
They include, for example, Microsoft Windows SKD Windows 3.1 and 
AVK/Windows 1.1. A suitable network program for providing, for 
example, the functions of networks 124, 224, 324, 424, 552 is 
available from Telsis Inc. Suitable video graphics software for 
providing the displays of the applications programs of the 
computer supported collaboration of the present invention include 
640x480 VGA mode. 

The programming for windows within video conferencing 
system 100 or computer supported collaboration system 100 in the 
preferred embodiment is minimal. Application programs 114, 120, 
214, 220, 314, 320, 414, 420 may open an all gray window which 
covers the upper half of their display devices. Two buttons may 
be created to provide operations previously described. One 
button may provide a telephone book dialog box and the other may 
hang up calls in order to terminate a conference. Two black 
rectangles may be drawn in the window for viewing images as also 
previously described . 

One window may be provided for display of the local 
user and the other window for display of the remote user. The 
user may select a name/number from a list of the telephone 
numbers and either press the CALL button to invoke the call, or 
press a cancel button, A message loop may be implemented, and if 
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a message is not available, the comiaunications driver is allowed 
to run. A timer message to this window is enabled at a rate 
equal to the frame rate. When the window receives this message 
it determines whether any data is ready to be sent or ready to be 
decompressed. A driver which may be used for the video/ audio 
compression/decompression is the Audio Visual Kemal for Windows 
available from Intel corporation. 

The setup of the audio visual kernal program can be 
broken up into three parts: the common setup, the playback 
setup, and the capture setup. In the common setup audio visual 
kernal starts by opening the audio/video device and registering 
the video teleconference window with audio visual kernal. This 
allows audio visual kernal to send messages to the window. A 
view may be created which maps a 256x24 0 bitmap onto the 64 0x480 
videos, graphics, audio (VGA) screen. The video data may be 
displayed in this view and chroma keyed with the VGA display. 

In the playback setup the first thing created is a 
playback group. This playback group allows control of streams 
that are created in this playback group. This control may 
include play, pause and flush. These streams are controlled and 
synchronized as previously described. Group buffer 556 is 
created to hold the compressed data wherein the compressed data 
includes both the audio and video data streams interleaved as 
also previously described. A video stream and an audio stream 
are created within the playback group buffer and the streams are 
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applied to their respective decompression engines 560, 564. A 
connector is created which copies and scales the decompressed 
video data to the view created in the common set up. 

In the capture setup a capture group is created. This 
capture group allows record, pause and flush control of streams 
that are created in this group. These streams are controlled and 
synchronized together. Group buffers 528, 532 are created, one 
for video input signals and one for audio input signals. Group 
buffers 528, 532 hold the compressed data. The compressed data 
for audio and video are stored separately in buffers 528, 532 and 
interleaved in video/audio interleaving device 544. 

The input /output for audio visual kernal may be divided 
into a capture section and a playback section. The capture 
section is called using a polling mechanism. Captured and 
compressed data is read into the memory of personal computers 
102, 104. This data movement is done for both the audio and 
video data streams which are already divided into frames. The 
capture section then combines a single frame of audio with a 
single frame of video to create a composite or interleaved frame. 

The size of the frame is checked with the communica- 
tions driver to see if it fits in the allocated bandwidth. If 
the size of the frame is too large, the capture section nulls out 
the video data. This involves setting the video data to pre- 
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determined values. These predetermined values tell the decoder 
to display the previous frame again. The visual result is a 
lovering of the frame rate. The composite frame header, video 
data, and audio data are then sent to the communications driver. 

The playback section is invoked by the communications 
driver when data is available. The playback section reads the 
data from the communications driver into host memory and then 
delivers it, as is. to the Action Media 11- card available from 
Intel Corporation. 
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CIAIMS 

1. A system for computer supported collaboration, 
comprising: 

first and second applications having respective 
first and second input devices for receiving first and second 
input data ; 

said first and second applications having means 
for providing first and second application data in accrrdance 
with said first and second input data respectively; 

network means coupled to said first and second 
applications for receiving said first and second application data 
and transmitting said first and second application data between 
said first and second applications; and, 

each of said first and second applications having 
means for receiving said transmitted data and processing both 
said first application data and said second application data. 

2. The system for computer supported collaboration of 
Claim 1,* wherein the same application data of said processed 
application data is processed simultaneously by said first and 
second applications. 
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